The statistical significance filter leads to overconfident expectations of replicability

نویسندگان

  • Shravan Vasishth
  • Andrew Gelman
چکیده

We show that publishing results using the statistical significance filter—publishing only when the p-value is less than 0.05—leads to a vicious cycle of overoptimistic expectation of the replicability of results. First, we show analytically that when true statistical power is relatively low, computing power based on statistically significant results will lead to overestimates of power. Then, we present a case study using 10 experimental comparisons drawn from a recently published metaanalysis in psycholinguistics (Jäger et al., 2017). We show that the statistically significant results yield an illusion of replicability. This illusion holds even if the researcher doesn’t conduct any formal power analysis but just uses statistical significance to informally assess robustness (i.e., replicability) of results.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

The illusion of power: How the statistical significance filter leads to overconfident expectations of replicability

We show that publishing results using the statistical significance filter—publishing only when the p-value is less than 0.05—leads to a vicious cycle of overoptimistic expectation of the replicability of results. First, we show through a simple derivation that when true statistical power is relatively low, computing power based on statistically significant results will lead to overestimates of ...

متن کامل

What Statistical Significance Means

Sohn (1998) presents a good argument that neither statistical significance nor effect size is indicative of the replicability of research results. His objection to the Bayesian argument is also succinct. However, his solution of the `replicability belief' issue is problematic, and his verdict that significance tests have no role to play in empirical research is debatable. The strengths and weak...

متن کامل

The Role of Statistical Significance Testing In Educational Research

The research methodology literature in recent years has included a full frontal assault on statistical significance testing. The purpose of this paper is to promote the position that, while significance testing as the sole basis for result interpretation is a fundamentally flawed practice, significance tests can be useful as one of several elements in a comprehensive interpretation of data. Spe...

متن کامل

The validity, diagnostic value and replicability of Bender Visual-Motor Gestalt Test in traumatic brain injury patients

Introduction: Bender Gestalt test is one of the most famous neuropsychological tests that is simple and it can be used to examine brain injuries. The objective of this research was to investigate the validity, diagnostic strength and the replicability of the Bender Visual-Motor Gestalt Test in patients with traumatic brain injury (TBI). Methods: 240 participants were tested in a case-control st...

متن کامل

Statistical Significance and Effect Size Reporting: Portrait of a Possible Future

The present paper comments on the matters raised regarding statistical significance tests by three sets of authors in this issue. These articles are placed within the context of contemporary literature. Next, additional empirical evidence is cited showing that the APA publication manual's "encouraging" effect size reporting has had no appreciable effect. Editorial policy will be required to aff...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2017